pOCiJHENT RESUME 



ED 306 305 



TM 013 238 



AUTHOR 
TITLE 

PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Clark, Sheldon B.? Boser, Judith A* 
Seeking Consensus on Empirical Characteristics of 
Effective Mail Questionnaires: A First Step» 
Mar 89 

3lp.? Paper presented at the Annual Meeting of the 
American Educational Research Association (San 
Francisco, CA, March 27-31, 1989) • 
Reports - Research/Technical (143) — 
Speeches/Conference Papers (150) — Tests/Evaluation 
Instruments (160) 

MF01/PC02 Plus Postage- 

*Check Lists; *Mail Surveys; Multiple Choice Tests; 
Quality Control; *Questionnaires; Test Construction; 
*Test Format; Test Items 

Consensus; Experts; item Position (Tests); Response 
Rates (Questionnaires); Test Instructions 



ABSTRACT 

A study was undertaken to develop a checklist of 
desirable characteristics of mail questionnaires. The checklist was 
to^ reflect some degree of consensus among experts in survey research 
cuid to be used as a general guide by novice questionnaire designers. 
A second objective was to take a first step toward development of an 
objective measure of quality of questionnaires. A review of the 
literature on the design of mail survey questionnaires indicated that 
seven categories were most essential in influencing response rate and 
quality of responses to mail questionnaires: (l) general appearance; 
(2) instructions; (3) choice of items; (4) order of items; (5) item 
format; (6) choice of response options; and (7) wording. A survey 
instrument based on the literature review was formulated. Six authors 
of books on survey research and six members of the American 
Educational Research Association's Special Interest Group on Survey 
Research in Education were surveyed; five of the former and six of 
the latter responded to the survey. Results are disappointing. The 
reluctance of the experts to indicate that any but the most 
fundamental of characteristics were applicable t*- all mail surveys 
underscores the often-stated principle that questionnaires should be 
tailored to the particular population being surveyed. Alternative 
approaches to future research are enumerated. Eleven data tables are 
included. The "Desirable Characteristics of Mail Questionnaires" 
Questionnaire is appended. (TJH) 
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Seeking Consensus on Empirical Characteristics of Effective Mail 
Questionnaires: A First Step 

Introduction 

Mail surveys are used fi:equendy, particularly in higher education institutions (Fuqua, 
Hartmaii, & Bro>yn, 1982).! According to Babbie (1973), "survey research is probably the 

best Icnpwn and most widely used research method in the social sciences today To 

some e}dtent» ever^ne in the United States at least has been affected by surveys" (p* i). 
Whilc Ae research instrument is only one compcment of the overall research endeavor, in 
mail eanreys the questionnaire takes on added importance* The potential respondent 
encounters it in isolation, with no interviewerpresent to encourage the respondent to 
partici^ue or to provide explanations. The individual must be motivated to complete the 
questionnaire, and the questionnaire must be designed to facilitate the respondents 
providing valid responses* *The task requu:^ of respondents must appear to be easy and 
attn:?*tivc. • . . Anything [respondents] particularly dislike about the layout, wording, or 
emphasis of the questions may deter tfiem (Homville, 1978, p. 127)." Anderson, Berdie, 
and NiebiAr (1986) concur, noting that "powly constracted iformats [(the physical 
arrangement of questions on the page)] influence not only response rates but also the 
quality of responses obtained" (p. 23). 

Objectives of Current Effort 

GuideHnes fOT the novice. A common occurrence for those who have reputations as 
knowledgeable, experienced, and/or successful survey researchers is to be approached by 
novice questionnaire-designers ar?4 asked for some general guidelines for developing 
questionnaires or to review a tentative survey instrument and provide feedback. Survey 
research literature includes a number of wOTks tiiat describe various authors* systems of 
^survey design that include the construction of the instrument itself, sampling 
considerations, choosing questions, cover letfcrerfoUo^ up procedures, etc. Frequently, 
aiitiiprs* recommendations seem to directed toward questionnaires in general, but upon 
closer reading it can be determined that they (the authors) are focusing on questionnaires 
used in telephone or face-to-face inteiviews rather than pape^-and-pencil instruments 



Ipuqua, D.R., Hartman, B.W., & Brown, D.F. (1982). Survey research in higher 
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completed by the respondent himsel^crself and ictumed through the mail Also in the 
Uteratuie is a pletfipia of studes dealing with various effects (e.g., on response rate, on 
compl^eness of responses, on turnaround time) of alternative forms of one or more 
elements of questionnaire design (e.g*, structured choice versus open-ended responses, 
variations in length, manifold style versus booklet style)* 

It might be difficult, however, to identify a simple checklist of widely agreed upon 
ch^cteristics of good mail questionnaires that would be helpful to the novice The total 
system coiK:ept seems to be too restrictive for this purpose: not only do the suggestions 
offered represent just a single point of view, but the guidelines niay be so specific that they 
are difiBcul: to geneniUze to a situaticm other tlian that for which they are illustrated The 
prd^leni with journal-type recomnjendatiohs is that they are likely to be too narrow in scope 
for the purpose described One objective of the present study is to develop a simple 
cheddist of d^irable characteristics of mail questionnaires (a) that reflects some degree of 
consensus among experts in survey research, and (b) that can be used as a general guide by 
novice questionnaire designers. 

Research tool: There arc numerous examples in the literature of inconsistencies in 
findings related to siffvey research. Many explanations have been offered for these 
apparent discrepancies, including basic differences in the studies with regard to such 
factors as tfie survey population and subject matter of the qu^tionnaire. Another source of 
uncontrolled variation that may affect dependent measures is the quality of the sur\'ey 
instrument itself, but ho acceptable measure of this quality cun-ently exists. If such a 
measure were available, it might be useful in interpreting tiie contradictory findings of 
ostensibly comparable stiidies. A second objective of the present study is to take a first 
step in developing an objective measure of the quality of a mail questionnaire. 

Method 

Instrument Development 

Significant journal and book sources that provide general guidelines for designing mail 
questionnaires were identified through an investigation of cunent books in print and ERIC 
listings. Those sources containing guidelines only for other survey techniques (e.g., 
telephone or face-to-fece interviews) were not included unless it could be determined that 



the recommended procedures were equally applicable for mail surveys* The final list of 
sources is attached* 

the ^pecSic r^ommendatibns from each of these books or articles were listed and 
categorized* OiUy those characteristics which were deemed desirable by several of the 
auttors M^ere retained; diose chao^teristics mentioned by only a few authors and those 
about which there.was disagreement were excluded from the list The remaining list of 
deskable charactenstics was edited to exclude redundancies and re-categorized 
independently by the two authois. The authors then compared iand discussed their 
respective lists, ultimately producing the 83 items and seven categories contained in the 
instrument Tliroughout thek^ discussions the authors recognized that there was more than 
one way in which items could be categorized, and that tiie categorization of items and the 
labels selected were somewhat arbitrary* An examination of a copy of the instrument 
(which is attached), showing the categories and items, should help clarify the conceptual 
basis for the categories* 

The final grouping of items resulted in seven categories requiring varying numbers of 
responses* The categories and their respective numbers of items (responses) are as 
follows: 



General Appearance 14 

Instructions 8 

Choice of Items 8 

Order of Items 15 

Item Format 16 

Choice.of Response Options 10 

Wording 12 



Once tiie characteristics had been selected, it was appare*" that some might be more 
imp<Ktant than others* It was decided to first determine whic** jharacteristics should be 
included (tiie focus of this study), then to focus on the relative importance of each 
characteristic* The respondents were asked to indicate, for each item, the extent to which 
the characteristic would be recommended for mail survey questionnaires using the 
following ratings: ALL (recommended for all mail questionnaires); SOME (recommended 
for some but not all mail questionnaires); or NONE (not recommended for mail 
qiiesticMinaires)* There was space following each section under a heading of "otiier" for the 
respondents to add other characteristics that they thought should have been included in tiiat 
section* 



Thc qucstipnnairc was photocopied and assembled in booklet format (7" by 8-1/2") 
uarig.tvtdihccts printed on both sides, 

foldi^^ collated, and saddle-stitched The front page served as a cover and the last (or 
eighth) page was reserved for comments^ Identical directions for responding to the items 
werc placed at the top of each of the six inside pages. 

Participants 

It was considered imp^tant that the participants in tlus study be knowledgeable and . 
experienced in simrcy research and represent various reseaivh environments. Six authors 
of books oh survey research (froni die attached bibliography) were invited to participate. 
Their publicarions conminod guidelines for the totad development of mail questionnaires 
(including wordings orf^ format or layout) and were not specific to a particular 
research emphasis (academia, pubUc opinion polls, marketing research). Six experienced 
practitiohere of survey research were selected from the membership of the American 
Educational Research Association's (AERA's) Special Interest Group on Survey Research 
in Education on the basis of their activities in the group and on their survey research 
background. 

Detailed background information was provided by ten of the eleven individuals who 
participated in die study. In addition to the five who were authors of books on survey 
methodology, each of the remaining six had made formal presentations on issues of survey 
research methodology at national professional conferences. Each of the five authors is in a 
leadership position in an organization which has a focus on survey research. Four of the 
odier participants are employed in postsecondary institutions in units that focus on research 
and/or evaluation. ~ 

The ten v/ho supplied background information had a: lassed a total of 172 years of 
experience in survey research, with individual experience varying from 7 to 40 years 
(median = 15.5 years). All ten had carried out surveys in 1988 and most considered it a 
typical year. These researchers had conducted firom 1 to 30 surveys themselves, as well as 
providing consultation on others. The research focus and the target population varied both 
within and across individuals. The major types of surveys were described as public 
opinion, needs assessitnerit, program evaluation/effectiveness, and institutional, consumer, 
and attitude studies. Target groups enumerated included the following: the general public; 
prpgram.participants; students; alumni; consumers; client grcups; various occupational 



groups, including professionals (e.g., judges, lawyers); and groups of employees within 
organizations (e,g-,^i5upe managers).. 

A copy of the instrument, an explanatoiy cover letter, and a pre-stamped reply 
envelope, werejnailed to each of twelve experienced survey researchers who iare well 
known in the field and/or who have been active members of AERA's SIG on Survey 
Rescaich;in Education. One fpilowrup reminder which included another copy of the 
instnuhcnt was mailed to each of the nohrcspondents approximately one month after the 
initial mailing. Only one of the twelve potential participants in this phase of the study did 
not respond, for a response rate of 92 percent. 

Results 

Response Patterns 

The characteristics included in tiie questionnaire consisted of those that were endorsed 
by at least several autiiors of the survey research literature examined, so it is not surprising 
tiiat most of the 913 possible responses (83 items X 1 1 respondents) were either in die 
ALL or SOME category (6rpercent and 33 percent, respectively). Only 4 percent of 
responses fell in the NONE category, and 2 percent were left blank. 

Response tendencies differed from one individual to miotiier. On the two extremes, 
one person selected die ALL category for 86 percent of the characteristics, while another 
selected ALL for only 41 percent of die characteristics. The median number of times a 
respondent chdose die ALL category was 51 times, or for 61 percent of the characteristics. 
The median for responses in die SOME category was 33 percent. Responses in the NONE 
category ranged from 0 to 12 percent across individuals, with the median being 4 percent. 

The individuals responding to the survey were five authors of books on survey 
research and six experienced, practitioners of survey research, so it was deemed inadvisable 
to make statistical comparisons in response patterns betvv'een the two groups. 



. . Experts' Assessment of Desirable Characteristics 



Gonfiisin^ items . Based on the hand-written comments of the respondents and/or their 
fdluie to resjpohd, several items were believed to have been confusing to at least some of 
tfe respcMideiits, Tablc l lists these items and the response distributions for each. These 
items are excluded from subsequent analyses. 

• Ghayacteristics of somewhat limited application . Despite the overwhelming tendency of 
respondents to indicate that the characteristics listed were applicable to at least some mail 
surveys, 2 or more of the 11 respondents felt that 7 of the characteristics should not apply 
to any. mail survey (See Table 2). 

Overall assessment of characteristics . Tables;3 thix)ugh 11 summarize the opinions of 
the.expcrts surveyed on the characteristics presented (excluding those in Table 1). The 
tables appear in order of decreasing applicability of the characteristics to all mail 
questionha^ That is, Table 3 contains those items that ^ of the experts felt should be 
characteristics of sSl ni^ surveys,,while Table 11 lists characteristics that less than 30 
percent of the responding experts judged to be desirable for aii niail surveys. 

Discussion 

In terms of the original objectives of the study, the results were disappointing. Upon 
reflection, however, they were not surprising. The reluctance of this group of experts to 
indicate that any but the most fundamental of characteristics were applicable to all moil 
surveys underscores the oft-stated principle that questionnaires be tailored to the particular 
population being surveyed This is an especially compelling point in this study, since only 
those characteristics for which there was general agreement (or at least lack of 
disagreement) among the authors considered were extracted from the literature. 

Many of the hand-written comments contained qualifiers or described exceptions to 
particular practices. More importantiy, these hand-written comments pointed out that the 
charactoistics presented in the questionnaire are of varying importance to the likelihood of 
success of a mail survey. This supports the yiew-as the authors had proposed in the cover 
letter-that at least one more step is necessary in order to establish the relative importance of 
each of these characteristics to the probable success of a mail survey. 



\ 



thgNgxtStgp 

Unfortunately, this study ialso reveals a lack of a substantial "core" of characteristics 
that are essential cto M mail questionnaiies* Recall that the authors* ultimate goals for the 
IcMig-term effort,;of which this study was conceived to be mi initial step, were to develop 
<a).a list of wideIyragrced-iq)on characteristics that.can be used as a checklist for novice 
questibnhaiie designers, and (b) an evaluatipn form that can be used as a means of 
comjparing the quality of instruments across research studies* The outcome of the present 
study suggests that at least two differentiappioaches could be considered as the possible 
next step. 

Broaden the universal applicability . One general approach would be to pare down the 
existing list to include only thqse characteristics that apply to all successful mail 
questionnaires; 'fhis would require revising the wording of many items and eliminating 
others al^gether. The advantage of this approach would be the broad applicability of the 
resultuig list The disadvantage is that the resulting list might contain too many items like 
those in fable 3, with which virtually nobody would disagree, but which, by themselves, 
offer very little guidance for the novice. 

Narrow the foqus > Another approach would be to try to address relevant factors that 
are inherent in the SOMSE responses and to develop several different lists of characteristics, 
pediaps, based on certain attributes of the target population, purpose and topic of the 
survey^ etc. Depending on how specifically one might define his/her frame of reference, it 
might then be possible-in addition to making a lengthier, more situation-specific list of 
desirable questionnaire characteristics-to broaden the scope of the list to cover other 
observable aspects of the survey methodology (e.g., cover letter, stamp, original and reply 
envelopes). The disadvantage of such an approach is the concomitant decrease in 
generalizability of the chai?.cteristics so identified and the resultant increase in the likelihood 
that a checklist or evaluation instrument would not be available for a given application. 

Validation. After a revised list is devised (using either of the approaches above), its 
quali^ and appropriateness could be tested using an approach similar to that used in the 
presitnt study (i.e., relying on ix panel of experts to establish face validity). A less direct, 
more overtly empirical apprpacth might entail identifying instruments that had been used in 
survey efforts with varying degrees of success. A determination could then be made about 
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how consistent the quality of the instruments (and possibly associated materials) used in 
tfiese'surveys^as reflected in the tentative list of desirable characteristics-is with the 
gauged success of the survey endeavor. Such an approach would require that a researcher 
pperationalizc the concept of "success," which is ^t to vary depending on such factors as 
the population, the topic or objectives of the questionnaire, and the resources used. 

Conclusion 

Survey research is certainly not a "clean" discipline; in fact, it is probably not a 
discipline at all. Mail questionnaires are used for a multitude of very different purposes and 
are targeted to populations ihat can vary widely on almost any dimension one might name. 
Perhaps it is unreasonable, therefore, to expect that there might be some underlying 
"triiths'-beyond those that are very obvious-which apply to all good mail surveys. This 
heterogeneity of purpose and of target population may mean that attempts to establish such 
global truths are likely to be fruitless. The authors would argue, nevertheless, that the 
effort to establish guidelines and rating tools should jjst be abandoned; they might perhaps 
be re-directed. The question should become "At what level of specificity (in terms of 
purpose and target population) should 'standards* be established for mail questionnaires?" 
The abundance of mail surveys that increasingly seem to flood the mail-and the 
astoundingly poor quality of some of them-suggest that a definite need exists to improve 
and/or assess (for purposes of comparison) the quality of the instruments used. Hopefully, 
the present study has shed some light on how this might be accomplished. 
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Table V. Questionnaire Items That Were Probably Confusing to Experts 
Surveyed 



No: Who Of Those R esponding 
Did Not % % % 



3 


1570 


cno7 




A. 10* The back page does not contain items 
but may be used for comments. 


2 


67% 


33% 


0% 


C.l. The respondent is capable of providing 
the answers (i.e., s/he has the 
information). 


2 


67% 


22% 


11% 


C.3. One item does not provide the answer 
to another item. 


2 


44% 


44% 


11% 


E,7. For closed-end items, response options 
arc prccoded. 


2 


67% 


33% 


0% 


E.i4.c. For checklists, column headings are 
presented horizontally. 



Notes: 

Potential confusion about the meaning of an item was based on the frequency 
and nature of hand-written comments. 



These items are excluded from the tables that follow. 
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Table 2. Characteristics of Mail Questionnaires Which More Than 15% of 
Responding Experts Indicated Should Not Apply to Mail vSurveys 



No. Who "Of Those Responding 
DidNot '% % % 
Respond ALL SOME NONE Characteristic 



0 


36% 


45% 


18% 


C.5.d. For items used for skip/filter/screcn 
purposes, items pertaining to only some of 
the respondents are indented beneath the 
filter question. 


1 


0% 


70% 


30% 


D.3, Open-ended items appear last. 


0 


36% 


45% 


18% 


D.7.C. Within a topic/content area, the 
items progress from least objectionable to 
most objectionable. 


0 


45% 


36% 


18% 


p If necessary, either sublettering (e.g., 
*ri., 4b. 4c) or numbering by sections (i.e., 
starting each section with item 1) is used to 
limit the apparent number of items. 


0 


45% 


27% 


27% 


E.5* If an item stem requires two or more 
lines, the second and subsequent lines are 
indented* 


d 


36% 


45% 


18% 


E.14.a. For checklists, if long, a line is 
skipped after every three to six items. 


0 


27% 


45% 


27% 


G.3.j* [Instrument does not contain] the 
word "questionnaire" or "checklist" in 
heading or text. 
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Tabic 3. Characteristics of Mail Questionnaires Which 100% of Responding 
Experts Indicated Should Apply to All Mail Surveys 



No. Who 
Did Not 
Respond 


Of Those Responding 
% % % ' 
ALL SOME NONE 


Characteristic 


0 


100% 


0% 


0% 


A.5. Printing does not bleed through 


0 


100% 


0% 


0% 


A.6. Type is clear and legible. 


0 


100% 


0% 


0% 


B.3.a. [Instructions] specify when to put a 
check mark and when to write in a 
response. 


0 


100% 


0% 


0% 


E.ll. There is adequate space for 

I vopUIIUI 11^. 


0 


100% 


0% 


0% 


E.14.b For checklists, column headings are 
carried over from one page to another. 


0 


100% 


0% 


0% 


F.l.g. Response options are appropriate for 
the item. 


0 


100% 


0% 


0% 


G.l. The choice of words is appropriate to 
the literacy level of the survey population. 


0 


100% 


0% 


0% 


G.3.d. [Items do not contain] double 
negatives . . . and/or response options. 
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Table 4. Characteristics of Mail Questionnaires Which 90%-99% of Responding 
Experts Indicated Should Apply to All Mail Surveys 



No. Who 

Did Not 
Respond. 



0 
0 

1 

0 

0 

0 

0 

0 

0 



Of Those Responding 
% % % 
ALL SOME NONE 



91% 9% 0% 
91% 9% 0% 



90% 
91% 
91% 
91% 
91% 
91% 
91% 



10% 
9% 
9% 
9% 
9% 
9% 
9% 



0% 



0% 



0% 
0% 
0% 
0% 
0% 



Characteristic 



A.2. Instrument looks easy to complete. 

A. ll. Appreciation for completing the 
form is expressed. 

B. 3.b. [Instructions] indicate whether 
multiple responses are allowed. 

C. 4. All items are essential and relevant to 
the purposes of the ' survey. 

C.S.b. Instructions [for skip/filter/screen 
items] are few and simple. 

E.3. Each item and its response options are 
on the same page. 

E. 9. Response options are close to the item 
stem. 

F. l.b. Response options are mutually 
exclusive. 

F.l.d. Response options do not contain more 
than one alternative that could be correct 
unless multiple responses are allowed. 



Tabic 5. Characteristics of Mail Questionnaires Which 80%-89% of Responding 
Experts Indicated Should Apply to All Mail Surveys 



No, Who Of Those Responding 
Did Not % % % 



0 


82% 


18% 


0% 


A.3. Margins are adequate; instrument 
doesn't look crowded. 


0 


82% 


18% 

• 


0% 


A.9.a, The front page (or cover) contains 
the study/instrument title, prominently 
displayed. 


0 


82% 


18% 


0% 


B.2. Instructions are brief. 


0 ' 


82% 


18% 


0% 


B.6. If items appear on both sides of the 
page, an indication is given that the 
instrument continues on the reverse side 
(e.g., "please turn over"). 


0 


. 82% 


18% 


0% 


C.5.C. [For items used for skip/fiUer/screen 
purposes,] instructions appear immediately 
after the response o^^tions. 


0 


82% 


18% 


0% 


D.l.d. The initial u-^ms are nonthreatening. 


0 


82% 


18% 


0% 


F.l.a. Response options exhaust all 
possibilities or include "other," 
"undecided;" or "neutral" category. 


0 


82% 


18% 


0% 


G.3.b. [There are no] "loaded" items (that 
use emotionally colored words). 


0 


82% 


18% 


0% 


G.3.C. [There are no items that contain an] 
assumption of an existing state of affairs 
(e.g., "Do you still. . ."). 



ERIC 
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Tabic 6. Characteristics of Mail Questionnaires Which 70%-79% of Responding 
Experts Indicated Should Apply to All Mail Surveys 



No. Who 



ula Not 

Respond 


OIL 

yo 

ALL 


SOME 


Of 
70 

NONE 


Characteristic 


0 


73% 


18% 


9% 


A.9.C. The front page (or cover) contains 
the name of the sponsor. 


0 


73% 


27% 


0% . 


B.S. The tone of the directions is polite 
(e.g., "please")* 


0 


73% 


27% 


0% 


C.2. Each Item seeks jus't one piece of 
information. 


0 


73% 


27% 


0% 


D.l.a. The initial items are clearly 
connected to the stated purpose of the 
survey. 


0 


73% 


27% 


0% 


D.l.b. The initial items are applicable to all 
members of the survey population. 


0 


73% 


27% 


0% 


D.l.e. The initial items are interesting. 


0 


73% 


27% 


0% 


D.8. Items that require recall are organized 
by logical time sequence. 


0 


73% 


27% 


0% 


F.l.f. Response options are brief. 
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Table 7; Characteristics of Mail Questionnaires Which 60%-69% of Responding 
Experts Indicated Should Apply to All Mail Surveys 



No, Who Of Those Responding 
Did Not % % % 

Respond ALL SOME NONE Characteristic 



1 


60% 


4,0% 


0% 


A.l. The title of the study/questionnaire is 
iiKciy' 10 appeal lo me survey popuiaiion. 


0 


64% 


36% 


0% 


A.7. Size, and style of type used for 
headings is consistent throughout the 
instrument. Consistency is also evident for 
iicms ana response options. 


0 


64% 


27% 


9% 


B.l. General instructions that apply to the 
entire instrument are provided at the 
beginning of the instrument. ^ 


0 


64% 


27% 


9% 


C.S.a. The use of [skip/filter/screxn items] 
is justified. 


0 


64% 


27% 


9% 


D.4. Classification or demographic 
iiiiumiailun lo ouiiwiicu ai me ena ui inc 
instrument unless needed for screening 
purposes. 


n 

u 


OH /O 


jO /O 


u /o 


1^.0. iiems wiin Similar conieni arc groupca 
together; within each content group, items 
with the same response format are 


0 


64% 


36% 


0% 


E.8. Response options are arranged 
vertically (or in columns if several 
consecutive items use the same response 
options). 


0 


64% 


36% 


0% 


E.IO. The space for responding to items is 
on the same side of the page throughout 
the instrument. 


0 


64% 


36% 


0% 


F.2.b. Items with Likeri-type response 
options use a balanced scale. 


0 


64% 


36% 


0% 


G.3.a. [Items do not contain] jargon, 
technical terms, or uncommon 
abbreviations. 
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Table 8. Characteristics of Mail Questionnaires Which 50%-59% of Responding 
Experts Indicated Should Apply to All Mail Surveys 



No: Who Of Those Responding 
•Did^Not % % % 

EfiSJUmd ALL SOME NONE Characteristic 



0 


55% 


,45% 


0% 


D.l.c. The initial items are easy. 


6 


55% 


45% 


0% 


D.7.b. Within a topic/content area, the 
items progress from most familiar to least 
familiar. 


1 


50% 


50% 


0% 


E.13. When ranking, the number of items 
to be rsuiked is limited (e s three best nnrf 
three worst). 


1 


50% 


50% 


0% 


G.2. Both sides of an issue (or neither side) 
are included in the item stem. 


0 


55% 


45% 


0% 


G.3.e. [Instrument does not contain] 
negatively worded items coupled with 
agree/disagree response format. 


0 


.55% 


45% 


0% 


G.3.g. [Items do not contain] "giveaway" 
words (e.g., "all"). 




IV-; ■-■ 










- 


I 


? 
i 


Table 9. 


Characteristics of Mail Questionnaires Which 40%-49% of Responding 
Experts Indicated Should Apply to All Mail Surveys 






No; Who 
Did Not 
Respond 


Of Those Responding 
% % % 
ALL SOME NONE 




i 


k" - ^* 

ft' ' - 


0 


45% 


45% 


9% 


A.4. Paper is white or light-colored with 
dark ink. 




r; ^ 


0 


45% 


55% 


0% 


A.8. There are not too many variations in 
size and style of type. 


• ■* 

'* 




0 


45% 


45% 


9% 


A.9.b, The front page (or cover) contains 
general directions. 


f 




0 


45% 


55% 


0% 


A.9.d. The front page (or cover) contains 
the address of the sponsor. 






1 


40% 


50% 


10% 


.B.3.C. Instructions provide guidance for 
expected length of open-ended responses. 




k * 
p 


0 


. 45% 


45% 


9% 


B.4. Instructions are visually different 
from the body of the instrument (e.g., in 
size and/or style of type). 


J 




0 


,45% 


55% 


0% 


E.l. Items are numbered with Arabic 
numerals. 


\ 




0 


45% 


36% 


18% 


E.2. If necessary, either sublettering (e.g., 
4a, 4b, 4c) or numbering by sections (i.e., 
starting each section with item 1) is used to 
limit the apparent number of items. 






0 


A C Of 

45% 


27% 


27% 


E.5. If an item stem requires two or more 
lines, the second and subsequent lines are 
indented. 


• 




0 


45% 


55% 


0% 


E.6. The respondent is asked to circle or 
underline responses already presented 
rather than write them on a blank. 






0 


45% 


55% 


0% 


E.12. Open-ended items are used sparingly. 




i 


0 


45% 


55% 


0% 


F.2.a. Items with Likert-type response 
options have an appropriately labeled 
midpoint. 






0 


45% 


45% 


9% 


G.3.i. [Items do not contain] vague 
terminology (e.g., "the country," "just," 
"fair," "you"). 


< 


t 
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\ablc 10. 



Ciiaract'iristics- of Mail Questionnaires Which 30%-39% of 
Respprdins Experts Indicated Should Apply to All Mail Surveys 



No. Who Of Those 'Responding 
Did Not % % %, 

RfiSBJmd ALL §0f4£ NONE Characteristic 



0 


36% 


46% 


18% 


C.S.d. iFor items used for skip/filier/screen 
purposes, items pertaining to only some of 
the respondents are indented beneath the 
filter question. 


0 


36% 


55% 


9% 


D.2. If there arc any sensitive or difficult 
items, they appear in the middle or near 
the end of the instrument, but not at the 
very end. 


0 


36% 


55% 


9% 


D.5. If reference is made to a previous item, 
that item appears on the same page or on 
the facing page. 


p 


36% 


45% 


18% 


D.7.C. Within a topic/content area, the 
items progress from least objectionable to 
most objectionable. 


0 


36% 


55% 


9% 


E.4. Statements or questions, rather than 
phrases, are used in collecting 
demographic information (e.g., "How old 
were you on your last birthday?" instead of 
"Age"). 


0 


36% 


45%- 


18% 


E.14.a. For checklists, if long, a line is 
skipped after every three to six items. 


0 


36% 


64% 


0% 


F.l.e. Response options include both sides 
of issue in question. 


0 


36% 


55% 


9% 


,G.3.f. (Items do not contain] qualifying 
clauses, especially at end of stem 
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Table 11. Characteristics of Mail Questionnaires Which Less Than 30% of 
Responding Experts Indicated Should Apply to All Mail Surveys 



No. Who Of Those Responding 
Did Not % % % 



1 
i 


V vo 


- I\J /o 


J\J /o 


g^/tvt i.Tnv 

\j*D* vypeu'-enaea iiems appear iesi. 


0 


11% 


64% 


9% 


DJ.a. Within a topic/content area, the 
items progress from general to specific* 


1 


10% 


80% 


10% 


D.v.d. Within a topic/content area, the 
items progress from objective to subjective. 


0 


9% 


91% 


0% 


F.lx. Include a "don*t know" option. 


0. 


18% 


82% 


0% 


F.3* Sensitive information (e*g., age, 
salary) is collected using ranges for 
response options* 


0 


18% 


73% 


9% 


G*3*h* [Items do not contain] inexact words 
or phrases (e.g., "any," "most," "several," 
"usually," "often," "regularly," "much the 
same"). 


0 


27% 


45% 


27% 


G.3.j* [Instrument does not contain] the 
word "questionnaire" or "checklist" in 
heading or text. 
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Note: The questionnaire is reduced in 



COMMBNTSi 



size for inclusion in this paper. 



DESIRABLE CHARACTERISTICS 



OF MAIL QUESTIONNAIRES 



iNSTMVCTiONS: Listed on the following pages are some generally agreed upon 
cliaracteristics of effective mail questionnaires. Please^ indicate tfte relative importance of 
each characteristic for mail survey quesfionnaires by circling your response to the right ^ 
the item on the following busts: 

ALL s= recommended for gt(m ail survey questionnaires 
SOifE = recommended for some b ut not all mail surveys 
SON^ = not recommended 



Space is provided at the end of each section for you to add any questionnaire 
cttnractenstics which are net listed but which you think should he included in a list of 
characteristics recommended for all mail survey questionnaires. 



SHELDON B* CLARK 
Oak Ridge Associated UnlversMfeii 

JUDITH A« BOSER 

Hiank ya for ihirUg y.nr enrertencc ani enperlUc with uf \n ihU rmarch. University of Tennessee 

Ptcafc rcUm U S**y Voscr, The VnUtrtMy of Tennessee, 212 CUilon, KnoKvUte, TN 37996. 
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^^ote : The questionnaire is reduced i, 



FUmi kuHute the nkthfe impcrUmc* of took ckr^pairuticfor mail swvey quesHon- 
^ mtk^Mbf ckMtyomr$tfOKt€ to the right 

^AtL m TiemmiUtif^^ smyey fuestioMokes 
SOME m f ic o mmmi idfpr Jgac W (dl maU suneyt 
^ SONB M Mti rgeommMded ' ■ - 

1. IhetiikofthettodyAiueiUoniMlieUlUcdyio'appealtothe 

survey population..^...... '. ALL SOME NONB 

t Instnimcnt looks easy to con^lete.... ^ ALL SOME NONE 

3. Martins arp a degua tt; I m tn wicnt doesn't look crowded....^ ALL SOME NONE 

4. ^Pspcr U white or lifht-Mlorcd with dirk ink ALL SOME NONB 

5. Prlntint docs not bleed thfough paper. ~ . ALL SOME NONE 

<. Type Is clear and jetfole;.-,..........;..........^ ^ ALL SOME NONB 

7. rSaeandstyleof type used for headings is combtent 
throufhout the Imfrurnent.< CohsittaKy is also evident 

for items and response options... ALL SOME NONE 

S., There are not too nvmy variations in size and style of type:.... ALL SOME NONE 

' 9. Hie front page (or coyer) oontalns: 

a. the study/lmirumcnt title, prominently displayed AIJ. SOME NONE 

b. general dlrecUoos. ALL SOME NONE 

c. the name of the sponsor ALL SOME NONE 

d. the address of the q>onsor... ALL SOME NONE 

10. The back page docs not conuln items but may be used for commcnU ALL SOME' NONE 

] ]. Appreciation for complctint the instrument is expressed ALL SOME NONE 



Size for inclusion in this paper. 



PUase indicate the relative Impcrtance cfeach characOristicfcr mail sirvey queMim^ 
nairee hy circWtg yom response to the right of the item on the following basis: 
ALL « recommended for aUmail sirvey ^[uestionnaires 
SOME s rtcommendedfcf tam, buitiotaU mail smveye 
NONE = not recommended 



n. fnstructioiM 

1 . Cenersl instructions that apply to the entire Instrument are 

provided at the bef inning of the instrument..... ALL SOME NONB 

2. Instructions are brief. ALL SOME NONE 

3. Instructions are clear 

a. They specify when to put « check mark and when to 

write in a response,...., ., ALL SOME NONE 

b. They indicate whether multiple responses are allowed. m ..^ ALL SOME NONE 

c Thr^ provide guidance for esipccted length of opcn-cndcd 

responses , ALL SOME NONE 

4 . Instructions are visually dtfTercnt from the body of the 

Instrument (e.g., in size and/or style of type). .,„., AIX SOME NONE 

5. Ihe tone of the directions Is polite (e.g.. "please"), „ „ ALL SOME NONE 

6. If items appear on both skies ofthe page, wiindicatiun is given 
that the instrument continues on t.^e reverse side 

(e.g., "please turn over") ALL SOME NONE 

Other 

C« rholfg of Urns 

1 . The reqxsndcnt is capable of providing the answers Q.e,, t/tie 

has the Informalloo) ALL SOME NONE 

2. Each item sedcs Just one {riece of information. .,, ALL SOME NONE 

3. One item does not provide the answer to another item ., ALL SOME NONE 

4. All items are esserilal and relevant to the pufposes of the survey. ALL SOME NONE 

5 . Vor items used for skip/niter/saeen purposes: 

a. The use of this type is justified ALL SOME NONE 

b. Instructions are few and simple m.» ALL SOME NONE 

c. L-uiructions appear immediately after the response options„,M ALL SOME NONB 

d. Items pertaining to only some of the respondents are 

indented beneath the niter question „ ALL SOME NONE 

OiIkt: 
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Note: The questionnaire is reduced 



rieme mdkmi the rekiive mfOrtmc€pfeack cf^octavtieffr mailswyey queftim- 
juMTtf by ckcSMt >mr rtsponde to the riihl of the item on thefothwint basis: 
ALL nrec6mmeti4edfor^maUsmr¥€yiiuestiM^ 

SOME xr^conmunM for iSBicbmlfM 

NOSk X Moi reeommenJ^ 



Order of Iteiiii 

1. Tibe initUI Ueam ai«: 

«. ckvlycomcledloUie ALL SOME NONE 

b. «pplic»b!e lo«l!meii4>as of Uie survey popuUUoa ALL SOME NONE 

c e«fy^ — Ali SOME NONE 

d nontbretienlng..... ALL SOME NONE 

c. intcretUaf ^ ALL SOME NONE 

2. If there are any aemkive or difllcuU items, ihcy appear in the 
middkccfievlliecndoribeiitttrun^ A!i SOME NONE 

3. Opcn^ed Uem* appear last. ^ ALL SOME NONE 

4 . Clusificatjcn or demographic infoanation is solicited at the 

end of the Imuuoienlunlea* needed for Sffcemng pu^ ALL SOME NONE 

5. Ifrefacnce is made to a previous item that item appears on the 

SOTie page or on the facing page^^ ALL SOME NONE 

6. Items with similar corto* are grouped together; within each 
contorx group, itona with the same re^Nxtsc formal are 

prcsenicd logciher.^ - ALL SOME NONE 

7. Within a topictoitcnt area, the itons progress from: 

a. general to spfcinc ^ Ali, SOME^NONE 

b. most fan^llar to least familiar.-.. — Ali- SOME NONE 

c least objectionable to most objectionable. Ali- SOME NONE 

d objective to subjective. — Ati SOME NONE 

S. Items that require recall are organized by logical lime sequence ALL SOME NONE 

Other 
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in size for inclusion in this paper. 



PUaseifidkiUthenl^iveimponmceofeachdiaraaniitkfortKulsio^ 
nairexbydrctiMt jomrtspowtotheritHoftheUemonthefolkmibasu: 
AIL « rec cm m tniti fortMjf^ nmy q u est iotwairtt 

SOME z recommended for JsaK ter moT «ff mail swveye 

NOSE = not recommended 





Hem Format 








1. 


Items are numbered with Arabic nunxrals 


ALL 


SOME 


NONE 


2. 


Ifnecesssiy, either fublettcriag (&g.,4a, 4b,4c) or mahbering 
by sections (jut^ starting each scctioo with Item 1) is used to 
limit the apparent number of items.. 


~ « ALL 


SOME 
SOME 

SOME 


NONE 
NONE 

NONE 


•% 
J. 








Statements or questions* rather than phfascs» are used in 

MwnMdf iKnw^nfxiic iniOttiMUOKi ^cg<, xiOw oio 

were you ofTyour last birthday?* instead of "Age"),.. 


ALL 


5. 


If an item stem req^ilrcs two or more line*, the second and 
subsequent lines are indented. 


ALL 


SOME 


NONE 


6. 


lite respondent is asked to drde or underline responses already 
presented rather than write them on a blank. ^ ........ 


ALL 


SOME 


NONE 


7. 


I^ closed-end ilems» response options are precoded....... 


ALL 


SOME 


NONE 


8. 


flicspvDc tfium* are wTwigou vciuciiiy ^or ut cocumns ii several 
consecutive items use the same response options) 


ALL 


SOME 


NONE 


9. 


Rc^onsc options are close to the item stem 


ALL 


SOME 


NONE 


10. 


IIk ^ce for re^^onding to items is on the same side of the 

page throughout the iiuirumenl..^ 


ALL 


SOME 


NONE 


II. 


There is adequate space for responding, .. 




SOME 


NONE 


12. 


Open-ended iums are used sparingly 


ALL 


SOME 
SOME 


NONE 
NONE 


i3. 


When ranking, the luriber of items to be ranked is limited 
(e.g„ three best and three worst)... ... 




14. 


It)r checklists: 










a. If long, a line is skipped alter evety three to six items... 


ALL 


SOME 


NONE 




b. Column headings are carried over from one page to another.... 


ALL 


SOME 


NONE 




c. Column heading! are presented horizonully 


ALL 


SOME 


NONE 



Otha: 
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Note: The questionnaire is reduced 



tkate i<<irnf» the fthtbtt imponmct ef each chmaatristic for nktit swyty qutMion- 
mkeskfcMki yomresfovetotkeriikteftheitpHOHthefoUowiAtbens^ 
Alls « TimmmnitifarmM mail amMj ^'rrti/mmiires ; 
SOkR * ra c oam mM far tsm, hamotaU mailsmveye 
:H0HR It met reco m mended. - 



F. Choice of Rtspopgg OpHain 

1. Roponte opiiam: 

a. calaustanpaBS>UtUesoriodtide*oibcr,**undeddcdr 



or '^netitrtl' c»tegory_ , — « « ^ Ali SOME NONE 

-b. ife motually neltaive...^ ALL SOME NONE 

' c include a "dont know" optioo..- Ali SOME NONE 

d do rwcointainnwre than one altciraUve that 

inkas tnulUple response* are allowed ALL SOME NONE 

t Include both lidest of Uaoc In question. ALL SOME NONE 

f. are brief:...^.^^^ ^ ^ „ Ali SOME NONE 

f. arc appropriate ^or the Item. ALL SOME NONE 

2. Items with Lakcrt-iype response options: 

a. iMve an appropriately labeled midpoint AIX SOME NONE 

b. use a balanced scale. .. ALL SOME NONE 

3. Sensitive inrom3aUon(e.g., age, salary) Is colkcicd using 

ranges for response options « „ AIX, SOME NONE 



Oiba: 
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size for inclusion in this paper. 



Flease indkoCt the rehlm imporianuofeach char aatriaic for maiistmy qiieMicn- 
moires by drcUtig your respomu to the rifht of the item on thefoUowini basis: 
AIL = ficomminded for alt mail swvey qitejtiomnaires 

SOME ac recomvmemdedfor sool but mot aU mail sw¥eye 

NONE = mot recommended 

G. Wording 

1. Thedioiceofwordsisapproprtateioibeliieracylevdoflbc 

tvTfZy populsUoO; ^ „. ALL SOME NONE 

2. Both aides of an Issue (or ndtber side) are included in the iiera stem.. ALL SOME NONE 

3. Items arn simple, diFect. and unanibifuout. Thev donot conuln 
instances of any of the foUowing pitfalls: 

a. Jargon, lechnical temia, or uncommon abbrevtau'ons ALL' SOME NONE 

b. 'Loaded* items (that use onotionaily colored words) ALL SOME NONE 

c Assumpu'onofanoisiingsuteofarrain 

(04., "Do you still...*) - » » ALL SOME NONE 

d Double negatives In items and/or 'csponse opu'ons. ALL SOME NONE 

c Ncgaitvdy worded items coupled with / veeAlisagree 

response format ^ A!i SOME NONE 

f. Qualifying clauses, especially at end of Sirm ALL SOME NONE 

g. "Giveaway" words (eig., "all")... ALL SOME NONE 

h- Inexact nconls or phrases (eg., "ariy," "most," several, ' 

"ususlly," "often," "regularly," "much the same"). ALL SOME NONE 

i. Vague terminology (e.g.» "the country," "just," "fair," 

"you") ~ ALL SOME NONE 

j. llie word "questionnaire" or "checklist** in heading or text. ALL SOME NONE 



Other 



(Please continue to page 8) 
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